install in dev mode is the easiest without having to publish it first

Aha, I see. Thanks for the tip

Thanks for the tips! I will take them for a spin

I can try taking it out see if it fix the issue. But i feel it is not the root cause

task = Task.init(
        project_name=PROJECT_NAME,
        task_name=TASK_NAME,
        task_type=Task.TaskTypes.data_processing,
    )
    task.set_repo(
        repo="git@xxx.git",
        branch=branch
    )
    task.set_base_docker(
        docker_image="docker-image",
    )
    task.execute_remotely(queue_name=QUEUE_NAME)

This is how I use the task init

PROJECT_NAME = "test"
TASK_NAME = "test_connect"
QUEUE_NAME = "default"
task = Task.init(project_name=PROJECT_NAME, task_name=TASK_NAME)

config = {
    "name": "foo",
    "arg1": "bar",
}
task.connect(config)

task.execute_remotely(queue_name=QUEUE_NAME)
# ------------- end of setup -------------

def dummy_op(config):
    pprint(config)

    return config


dummy_op(config)

Sreenshot also provided to show what "edit" button only appear in user property not hyperparamter
![image](...

I gave this it a try to switch from Task.create() to Task.init(). I think I am pretty close to switch to using init(). But still have issue of ModuleNotFoundError: No module named 'src' when using task.init().

My project setup look like this:

project_root/
    |--src/
    |--runbooks/
        |--run_task.py

So if I use Task.create(repo=xx, script="runbooks/run_task.py"), it works but if I switch to using Task.init() with the same repo setup (task.set_repo, and then follow by ...

I used these setup to load a pretty big dataset from s3:

dataset.add_external_files(
            source_url
        )
        dataset.upload(
            verbose=verbose
        )
        dataset.finalize()

But then seeing error complain about dataset doesn't exist. So my best guess is that the uploading is still happening in the background while the code has move forward to try to do something with that dataset.

So I am questioning if I have to explicitly add some logic to wait f...

that is just to compare the same functionality used in task.create() can be achieved when using task.init workflow. Not technically required

Hi John, exactly. If I have the model_id, I want to find its project. Do you have tip to do it in both way?

yes

And how would I do that? "check apiserver logs"

function's signature in the source code

Roughly I am trying to do this:

def upload_clearml_dataset_from_external_source(
    source_url,
    dataset_name: str,
    dataset_project: str,
):
    # reference: 

    dataset = Dataset.create(dataset_name=dataset_name, dataset_project=dataset_project)
    dataset.add_external_files(source_url=source_url)
    dataset.upload()

    dataset.finalize()


upload_clearml_dataset_from_external_source("

", name, project)

Dataset.get(dataset_project=project, dataset...

Thanks!

Aha I see. that works to retrieve the project id.

A side question, I notice in the clearml design, Task, Model, Dataset those object all have its base model, but not project concept. In terms of project, is there a quick way to retrieve project name based on its id?

m = Model(model_id)
print("Model.id:", m.id) # <- this is returning model_id, but I need project_id or project_name 

print("Model.data:", m.data) # <- AttributeError: 'Model' object has no attribute 'data'

Thanks, that works

Would it cause problem to manually set repo when using task.init()?

Thanks John! This is exactly it! need to reset the task first then the edit feature will show up

Awesome, thanks for the clarification!

Ok, thanks for the guidance.

The ultimate goal is to make sure run.py get run under project root directory because the underlying code has this assumption when writing it. Locally pycharm take care of this, but when execute remotely, need to take care of this otherwise it will complain Module Not Found .

A typical project setup I am working with look like this:

project_root/
    |--src/
        |--utilities.py
        |--foo.py
    |--runbooks/
        |--run.py

And typical import look like this, for ...

I see. Is there anything I can check from the user perspective?

Actually I am not sure. for enterprise users. Is it most commonly self-hosted?

Thanks! I will give this a try

Sure, essentially my local python project organized using "src layout", look like this:

foo/
    |--src/
        |--module.py
    |--pyproject.toml
    |--clearml_tasks/
        |--task1.py

in the project, it would use absolute import like from foo import module , and I would install foo project in a editable mode during setup.

When I trying to create clearml task and send it to remote server using above way (leverage requirements.txt to configure library dependencies, and pro...

Hi @<1523701070390366208:profile|CostlyOstrich36> , I tried out Task.add_requirements way to add packages, but it doesn't seem to be working as I expected. here is the snippet i used to setup this up:

Task.force_store_standalone_script()
    add_packages = ["fastparquet"]
    for pkg in add_packages:
        Task.add_requirements(pkg)
    task = Task.init(project_name=project_name, task_name=task_name)
    task.set_base_docker(docker_arguments="--env CLEARML_AGENT_SKIP_PYTHON_ENV_INSTA...

ignore the indentation, didn't align when I copy over

Hi John, the dataset.squash doc says "If a set of versions are given it will squash the versions diff into a single version", I want to double check will it only keep the latest version?

Because I don't want any old version stuff even old version have more stuff than the latest version